Speeding up Gibbs sampling by variable grouping

Gibbs sampling is a widely applicable inference technique that can in principle deal with complex multimodal distributions. Unfortunately, it fails in many practical applications due to slow convergence and abundance of local minima. In this paper, we propose a general method of speeding up Gibbs sampling in probabilistic models. The method works by introducing auxiliary variables which represent assignments of the original model variables to groups. Our experiments indicate that the groups converge early in the sampling. After they have converged, the original variables no longer need to be sampled, and it becomes possible to resample an entire group at a time, greatly speeding up the sampler. The proposed ideas are illustrated on LDA and are applicable to many other topic models.